Towards Everyday Virtual Reality through Eye Tracking

Durch Entwicklungen in den Bereichen Computergrafik, Hardwaretechnologie, Perception Engineering und Mensch-Computer Interaktion, werden Virtual Reality und virtuelle Umgebungen immer mehr in unser tägliches Leben integriert. Head-Mounted Displays werden jedoch im Vergleich zu anderen mobilen Geräten, wie Smartphones und Smartwatches, noch nicht so häufig genutzt. Mit zunehmender Nutzung dieser Technologie und der Gewöhnung von Menschen an virtuelle Anwendungsszenarien ist es wahrscheinlich, dass in naher Zukunft ein alltägliches Virtual-Reality-Paradigma realisiert wird. Im Hinblick auf die Kombination von alltäglicher Virtual Reality und Head-Mounted-Displays, ist Eye Tracking eine neue Technologie, die es ermöglicht, menschliches Verhalten in Echtzeit und nicht-intrusiv zu messen. Bevor diese Technologien in großem Umfang im Alltag eingesetzt werden können, müssen jedoch noch zahlreiche Aspekte genauer erforscht werden. Zunächst sollten Aufmerksamkeits- und Kognitionsmodelle in Alltagsszenarien genau verstanden werden. Des Weiteren sind Maßnahmen zur Wahrung der Privatsphäre notwendig, da die Augen mit visuellen biometrischen Indikatoren assoziiert sind. Zuletzt sollten anstelle von Studien oder Anwendungen, die sich auf eine begrenzte Anzahl menschlicher Teilnehmer mit relativ homogenen Merkmalen stützen, Protokolle und Anwendungsfälle für eine bessere Zugänglichkeit dieser Technologie von wesentlicher Bedeutung sein. In dieser Arbeit wurde unter Berücksichtigung der oben genannten Punkte ein bedeutender wissenschaftlicher Vorstoß mit drei zentralen Forschungsbeiträgen in Richtung alltäglicher Virtual Reality unternommen. Menschliche visuelle Aufmerksamkeit und Kognition innerhalb von Virtual Reality wurden in zwei unterschiedlichen Bereichen, Bildung und Autofahren, erforscht. Die Forschung im Bildungsbereich konzentrierte sich auf die Auswirkungen verschiedener Manipulationen im Klassenraum auf das menschliche Sehverhalten, während die Forschung im Bereich des Autofahrens auf sicherheitsrelevante Fragen und Blickführung abzielte. Die Nutzerstudien in beiden Bereichen zeigen, dass Blickbewegungen signifikante Implikationen für diese alltäglichen Situationen haben. Der zweite wesentliche Beitrag fokussiert sich auf Privatsphäre bewahrendes Eye Tracking für Blickbewegungsdaten von Head-Mounted Displays. Dies beinhaltet Differential Privacy, welche zeitliche Korrelationen von Blickbewegungssignalen berücksichtigt und Privatsphäre wahrende Blickschätzung durch Verwendung eines auf randomisiertem Encoding basierenden Frameworks, welches Augenreferenzunkte verwendet. Die Ergebnisse beider Arbeiten zeigen, dass die Wahrung der Privatsphäre möglich ist und gleichzeitig der Nutzen in einem akzeptablen Bereich bleibt. Wenngleich es bisher nur wenig Forschung zu diesem Aspekt von Eye Tracking gibt, ist weitere Forschung notwendig, um den alltäglichen Gebrauch von Virtual Reality zu ermöglichen. Als letzter signifikanter Beitrag, wurde ein Blockchain- und Smart Contract-basiertes Protokoll zur Eye Tracking Datenerhebung für Virtual Reality vorgeschlagen, um Virtual Reality besser zugänglich zu machen. Die Ergebnisse liefern wertvolle Erkenntnisse für alltägliche Nutzung von Virtual Reality und treiben den aktuellen Stand der Forschung in mehrere Richtungen voran.With developments in computer graphics, hardware technology, perception engineering, and human-computer interaction, virtual reality and virtual environments are becoming more integrated into our daily lives. Head-mounted displays, however, are still not used as frequently as other mobile devices such as smart phones and watches. With increased usage of this technology and the acclimation of humans to virtual application scenarios, it is possible that in the near future an everyday virtual reality paradigm will be realized. When considering the marriage of everyday virtual reality and head-mounted displays, eye tracking is an emerging technology that helps to assess human behaviors in a real time and non-intrusive way. Still, multiple aspects need to be researched before these technologies become widely available in daily life. Firstly, attention and cognition models in everyday scenarios should be thoroughly understood. Secondly, as eyes are related to visual biometrics, privacy preserving methodologies are necessary. Lastly, instead of studies or applications utilizing limited human participants with relatively homogeneous characteristics, protocols and use-cases for making such technology more accessible should be essential. In this work, taking the aforementioned points into account, a significant scientific push towards everyday virtual reality has been completed with three main research contributions. Human visual attention and cognition have been researched in virtual reality in two different domains, including education and driving. Research in the education domain has focused on the effects of different classroom manipulations on human visual behaviors, whereas research in the driving domain has targeted safety related issues and gaze-guidance. The user studies in both domains show that eye movements offer significant implications for these everyday setups. The second substantial contribution focuses on privacy preserving eye tracking for the eye movement data that is gathered from head-mounted displays. This includes differential privacy, taking temporal correlations of eye movement signals into account, and privacy preserving gaze estimation task by utilizing a randomized encoding-based framework that uses eye landmarks. The results of both works have indicated that privacy considerations are possible by keeping utility in a reasonable range. Even though few works have focused on this aspect of eye tracking until now, more research is necessary to support everyday virtual reality. As a final significant contribution, a blockchain- and smart contract-based eye tracking data collection protocol for virtual reality is proposed to make virtual reality more accessible. The findings present valuable insights for everyday virtual reality and advance the state-of-the-art in several directions

Eye-tracking devices for virtual and augmented reality metaverse environments and their compatibility with the European Union general data protection regulation

Even though the Metaverse from science fiction is not a reality yet, it is possible to take a glimpse into how it might look like. Several companies and platforms are currently working and developing their versions of how the Metaverse will look in a not-so-distant future. However, the current vision of the Metaverse does not only encompass software. A great deal of companies is complementing their Metaverse projects with Virtual and Augmented Reality devices such as headsets and glasses. In this line, one of the last technological advancements in virtual and augmented reality devices included the introduction of eye-tracking technology. However, when new and additional kinds of data are processed, emerging risks for data protection might arise. While there is an already incipient stream of scholarship that analyses the risks that eye-tracking devices might entail for privacy, such literature mostly focuses on the technical side. However, no scholarship, up to this moment, has examined such devices from a legal perspective and particularly from a data protection lens. This paper will, therefore, discuss the compatibility of eye-tracking devices for virtual and augmented reality environments with the European Union General Data Protection Regulation (GDPR). Being the GDPR considered a worldwide role model in terms of fundamental rights protection, the compatibility of such devices with one of the most severe data protection regimes will be put to the hardest test. The paper will do so by analyzing the state of the art of the technology, its use in headsets and glasses for virtual and augmented reality Metaverse environments, and the potential risks that such use might entail for data protection. After that, such risks will be confronted with the relevant applicable provisions of the GDPR. Finally, the paper will issue policy recommendations regarding the need for more interdisciplinary research on privacy-enhancing techniques to solve the privacyutility conundrum; careful monitoring of access to data by third parties, including data security and minimization requirements; more guidance from supranational Data Protection Authorities and more attention when designing privacy policies, especially for children

FakeNewsPerception: an eye movement dataset on the perceived believability of news stories

Extensive use of the internet has enabled easy access to many different sources, such as news and social media. Content shared on the internet cannot be fully fact-checked and, as a result, misinformation can spread in a fast and easy way. Recently, psychologists and economists have shown in many experiments that prior beliefs, knowledge, and the willingness to think deliberately are important determinants to explain who falls for fake news. Many of these studies only rely on self-reports, which suffer from social desirability. We need more objective measures of information processing, such as eye movements, to effectively analyze the reading of news. To provide the research community the opportunity to study human behaviors in relation to news truthfulness, we propose the FakeNewsPerception dataset. FakeNewsPerception consists of eye movements during reading, perceived believability scores, questionnaires including Cognitive Reflection Test (CRT) and News-Find-Me (NFM) perception, and political orientation, collected from 25 participants with 60 news items. Initial analyses of the eye movements reveal that human perception differs when viewing true and fake news

Exploring Human Perception while Reading Fake and Real News Articles

With the increased spread of misinformation on online platforms and the popularity of AI-generated text, there is a critical need to detect human perception regarding the truthfulness of news. Users’ believability in a news item influences the reading and sharing of that news. Hence, in order to reduce the spread of fake news online, it is important to understand how users\u27 engagement with fake and real news and users\u27 perceived believability impact their behavioral and physiological factors. In this work, we study human eye movements based on the truthfulness of news and their perceived believability. Using the publicly available FakeNewsPerception eye-tracking dataset, we investigate the relationship between the visual scanning behavior, distribution of attention over Areas of Interest (AOIs), and cognitive load with respect to truthfulness and perceived believability of news content using advanced gaze measures, such as Coefficient K, Gaze Transition Entropy, and Low/High Index of Pupillary Activity (LHIPA). Coefficient K is derived from eye movements and acts as a dynamic indicator of fluctuation between ambient/focal visual scanning behavior. The gaze transition entropy is a measure of predictability in Areas of Interest (AOI) transitions and the overall distribution of attention over AOIs. The LHIPA is an eye-tracked measure of pupil diameter oscillation that was introduced as an indicator of cognitive load. We observed the participants exhibit more ambient processing in visual scanning when they are unsure of the truthfulness of news and more focal processing when they do not believe the news is real. We found that the ambient/focal viewing pattern changes over time are similar for all the groups of participants we considered. Further, we observed a similar distribution of attention over AOIs and similar cognitive load among participants regardless of the truthfulness of the news and their believability of the news.https://digitalcommons.odu.edu/gradposters2023_sciences/1024/thumbnail.jp

Reinforcement learning for the privacy preservation and manipulation of eye tracking data

In this paper, we present an approach based on reinforcement learning for eye tracking data manipulation. It is based on two opposing agents, where one tries to classify the data correctly and the second agent looks for patterns in the data, which get manipulated to hide specific information. We show that our approach is successfully applicable to preserve the privacy of the subjects. For this purpose, we evaluate our approach iteratively to showcase the behavior of the reinforcement learning based approach. In addition, we evaluate the importance of temporal, as well as spatial, information of eye tracking data for specific classification goals. In the last part of our evaluation, we apply the procedure to further public data sets without re-training the autoencoder or the data manipulator. The results show that the learned manipulation is generalized and applicable to unseen data as well